The Proof of McAllester’s PAC-Bayesian Theorem

Extends McAllester’s PAC-Bayesian theorem [4], allowing for other tail bounds than Hoeffding’s, and simplifies the proof somewhat. 1 McAllester’s PAC-Bayesian theorem McAllester’s PAC-Bayesian theorem is an astonishing result. Although its proof requires only elementary techniques, it can be used to obtain very tight generalization error bounds for the Gibbs versions of (approximate) Bayesian classification models. In special cases, which are often practically relevant, we can obtain bounds for the corresponding Bayes versions as well. Suppose we are given a concept space C and a space of labeled examples X . We denote concepts by c ∈ C, examples by x ∈ X . Suppose also we are given a loss function l(c, x), mapping pairs (c, x) to the interval [a, b] or [a, b) (here, 0 ≤ a < b). There is an unknown data distribution D from which examples x can be drawn, and we do not make any assumptions on D. For each concept c, we can define the expected loss l(c) = E[l(c, x)], where the expectation is taken over x ∼ D. Given a sample S = {x1, . . . , xm} of size m, drawn independently and identically distributed (i.i.d.) from D, we can also define the empirical loss l̂(c) = m ∑ i l(c, xi). Note that ES[l̂(c)] = l(c). Depending on the loss function, it is possible to bound the probability (over S) for large deviations between l̂(c) and its expected value l(c). McAllester’s technique takes such a large deviation bound for single concepts c and transforms it into a bound for Gibbs concepts, which in turn depend on a distribution over all concepts in C (this is introduced in detail below). Therefore, we need to assume the availability of a large deviation bound for single l̂ = l̂(c), l = l(c). Namely, there have to exist constants C > 0, β > 0 and a function φ on (a, b)× (a, b) s.t. Pr {